Convolutional Neural Networks

Project: Write an Algorithm for Landmark Classification

In this notebook, some template code has already been provided for you, and you will need to implement additional functionality to successfully complete this project. You will not need to modify the included code beyond what is requested. Sections that begin with '(IMPLEMENTATION)' in the header indicate that the following block of code will require additional functionality which you must provide. Instructions will be provided for each section, and the specifics of the implementation are marked in the code block with a 'TODO' statement. Please be sure to read the instructions carefully!

Note: Once you have completed all the code implementations, you need to finalize your work by exporting the Jupyter Notebook as an HTML document. Before exporting the notebook to HTML, all the code cells need to have been run so that reviewers can see the final implementation and output. You can then export the notebook by using the menu above and navigating to File -> Download as -> HTML (.html). Include the finished document along with this notebook as your submission.

In addition to implementing code, there will be questions that you must answer which relate to the project and your implementation. Each section where you will answer a question is preceded by a 'Question X' header. Carefully read each question and provide thorough answers in the following text boxes that begin with 'Answer:'. Your project submission will be evaluated based on your answers to each of the questions and the implementation you provide.

Note: Code and Markdown cells can be executed using the Shift + Enter keyboard shortcut. Markdown cells can be edited by double-clicking the cell to enter edit mode.

The rubric contains optional "Stand Out Suggestions" for enhancing the project beyond the minimum requirements. If you decide to pursue the "Stand Out Suggestions", you should include the code in this Jupyter notebook.

Why We're Here

Photo sharing and photo storage services like to have location data for each photo that is uploaded. With the location data, these services can build advanced features, such as automatic suggestion of relevant tags or automatic photo organization, which help provide a compelling user experience. Although a photo's location can often be obtained by looking at the photo's metadata, many photos uploaded to these services will not have location metadata available. This can happen when, for example, the camera capturing the picture does not have GPS or if a photo's metadata is scrubbed due to privacy concerns.

If no location metadata for an image is available, one way to infer the location is to detect and classify a discernible landmark in the image. Given the large number of landmarks across the world and the immense volume of images that are uploaded to photo sharing services, using human judgement to classify these landmarks would not be feasible.

In this notebook, you will take the first steps towards addressing this problem by building models to automatically predict the location of the image based on any landmarks depicted in the image. At the end of this project, your code will accept any user-supplied image as input and suggest the top k most relevant landmarks from 50 possible landmarks from across the world. The image below displays a potential sample output of your finished project.

Sample landmark classification output

The Road Ahead

We break the notebook into separate steps. Feel free to use the links below to navigate the notebook.

Step 0: Download Datasets and Install Python Modules

Note: if you are using the Udacity workspace, YOU CAN SKIP THIS STEP. The dataset can be found in the /data folder and all required Python modules have been installed in the workspace.

Download the landmark dataset. Unzip the folder and place it in this project's home directory, at the location /landmark_images.

Install the following Python modules:

Step 1: Create a CNN to Classify Landmarks (from Scratch)

In this step, you will create a CNN that classifies landmarks. You must create your CNN from scratch (so, you can't use transfer learning yet!), and you must attain a test accuracy of at least 20%.

Although 20% may seem low at first glance, it seems more reasonable after realizing how difficult of a problem this is. Many times, an image that is taken at a landmark captures a fairly mundane image of an animal or plant, like in the following picture.

Bird in Haleakalā National Park

Just by looking at that image alone, would you have been able to guess that it was taken at the Haleakalā National Park in Hawaii?

An accuracy of 20% is significantly better than random guessing, which would provide an accuracy of just 2%. In Step 2 of this notebook, you will have the opportunity to greatly improve accuracy by using transfer learning to create a CNN.

Remember that practice is far ahead of theory in deep learning. Experiment with many different architectures, and trust your intuition. And, of course, have fun!

(IMPLEMENTATION) Specify Data Loaders for the Landmark Dataset

Use the code cell below to create three separate data loaders: one for training data, one for validation data, and one for test data. Randomly split the images located at landmark_images/train to create the train and validation data loaders, and use the images located at landmark_images/test to create the test data loader.

All three of your data loaders should be accessible via a dictionary named loaders_scratch. Your train data loader should be at loaders_scratch['train'], your validation data loader should be at loaders_scratch['valid'], and your test data loader should be at loaders_scratch['test'].

You may find this documentation on custom datasets to be a useful resource. If you are interested in augmenting your training and/or validation data, check out the wide variety of transforms!

Question 1: Describe your chosen procedure for preprocessing the data.

Answer:

  1. My code randomly resizes and crops the images to a 224x224 pixel square. A resize operation will simulate photos taken at different distances from the landmark. The random crop location will allow the CNN to recognize landmarks in the event the landmark is not fully in view in the input image. The 224x224 size was chosen to control for as many variables as possible when comparing the performance of a from-scratch CNN vs. the transfer learning CNN from the next step. The ResNet50 model I used for transfer learning in the next step was pre-trained on 224x224 images.

  2. Yes, I added a small degree of random rotation to images up to 2 degrees. The intention was to simulate unsteady hands while photographing. Flips and translations (aside from the implied translation from a random resize crop) were not included. Flips because landscape photos wouldn't typically be horizontally or vertically flipped, unlike say a selfie, which could be a mirror image or flipped left to right.

(IMPLEMENTATION) Visualize a Batch of Training Data

Use the code cell below to retrieve a batch of images from your train data loader, display at least 5 images simultaneously, and label each displayed image with its class name (e.g., "Golden Gate Bridge").

Visualizing the output of your data loader is a great way to ensure that your data loading and preprocessing are working as expected.

Initialize use_cuda variable

(IMPLEMENTATION) Specify Loss Function and Optimizer

Use the next code cell to specify a loss function and optimizer. Save the chosen loss function as criterion_scratch, and fill in the function get_optimizer_scratch below.

(IMPLEMENTATION) Model Architecture

Create a CNN to classify images of landmarks. Use the template in the code cell below.

Question 2: Outline the steps you took to get to your final CNN architecture and your reasoning at each step.

Answer:

I reused the model defined in the Lesson 4 transfer learning exercise solutions. Since this model was proposed for identifying objects like cars, horses, birds, etc, I figured it would generalize for mountains, buildings, and other landmarks as well

The initial training run using 100 epochs provided 24% accuracy.

(IMPLEMENTATION) Implement the Training Algorithm

Implement your training algorithm in the code cell below. Save the final model parameters at the filepath stored in the variable save_path.

(IMPLEMENTATION) Experiment with the Weight Initialization

Use the code cell below to define a custom weight initialization, and then train with your weight initialization for a few epochs. Make sure that neither the training loss nor validation loss is nan.

Later on, you will be able to see how this compares to training with PyTorch's default weight initialization.

(IMPLEMENTATION) Train and Validate the Model

Run the next code cell to train your model.

(IMPLEMENTATION) Test the Model

Run the code cell below to try out your model on the test dataset of landmark images. Run the code cell below to calculate and print the test loss and accuracy. Ensure that your test accuracy is greater than 20%.

Step 2: Create a CNN to Classify Landmarks (using Transfer Learning)

You will now use transfer learning to create a CNN that can identify landmarks from images. Your CNN must attain at least 60% accuracy on the test set.

(IMPLEMENTATION) Specify Data Loaders for the Landmark Dataset

Use the code cell below to create three separate data loaders: one for training data, one for validation data, and one for test data. Randomly split the images located at landmark_images/train to create the train and validation data loaders, and use the images located at landmark_images/test to create the test data loader.

All three of your data loaders should be accessible via a dictionary named loaders_transfer. Your train data loader should be at loaders_transfer['train'], your validation data loader should be at loaders_transfer['valid'], and your test data loader should be at loaders_transfer['test'].

If you like, you are welcome to use the same data loaders from the previous step, when you created a CNN from scratch.

(IMPLEMENTATION) Specify Loss Function and Optimizer

Use the next code cell to specify a loss function and optimizer. Save the chosen loss function as criterion_transfer, and fill in the function get_optimizer_transfer below.

(IMPLEMENTATION) Model Architecture

Use transfer learning to create a CNN to classify images of landmarks. Use the code cell below, and save your initialized model as the variable model_transfer.

Question 3: Outline the steps you took to get to your final CNN architecture and your reasoning at each step. Describe why you think the architecture is suitable for the current problem.

Answer:

Since the VGG16 model was used for transfer learning in an exercise earlier in the CNN course, I wanted to try a different architecture. I trained a ResNet50 using transfer learning. To do this, only the fully connected layer at the end of the network was updated, in order to map the output features from the resnet50 network to the number of output classes provided in the training dataset. ResNet50 offered a good compromise between FLOPS needed for training and overall accuracy, based on this arxiv article I found: Benchmark Analysis of Representative Deep Neural Network Architectures https://arxiv.org/pdf/1810.00736.pdf

(IMPLEMENTATION) Train and Validate the Model

Train and validate your model in the code cell below. Save the final model parameters at filepath 'model_transfer.pt'.

Original Training run, when model.parameters() was passed into the optimizer, instead of model.fc.parameters():

Epoch 1/20
Epoch: 1 Training Loss: 14557.629360 Validation Loss: 3183.475996
Validation loss decreased (inf --> 3183.475996). Saving model ...
Epoch 2/20
Epoch: 2 Training Loss: 10831.356453 Validation Loss: 2230.182967
Validation loss decreased (3183.475996 --> 2230.182967). Saving model ...
Epoch 3/20
Epoch: 3 Training Loss: 7867.816956 Validation Loss: 1729.451578
Validation loss decreased (2230.182967 --> 1729.451578). Saving model ...
Epoch 4/20
Epoch: 4 Training Loss: 6278.636573 Validation Loss: 1507.198152
Validation loss decreased (1729.451578 --> 1507.198152). Saving model ...
Epoch 5/20
Epoch: 5 Training Loss: 5293.944752 Validation Loss: 1295.387159
Validation loss decreased (1507.198152 --> 1295.387159). Saving model ...
Epoch 6/20
Epoch: 6 Training Loss: 4718.517012 Validation Loss: 1234.325491
Validation loss decreased (1295.387159 --> 1234.325491). Saving model ...
Epoch 7/20
Epoch: 7 Training Loss: 4238.656453 Validation Loss: 1133.867814
Validation loss decreased (1234.325491 --> 1133.867814). Saving model ...
Epoch 8/20
Epoch: 8 Training Loss: 3826.112483 Validation Loss: 1143.746305
Epoch 9/20
Epoch: 9 Training Loss: 3793.398960 Validation Loss: 1115.092531
Validation loss decreased (1133.867814 --> 1115.092531). Saving model ...
Epoch 10/20
Epoch: 10 Training Loss: 3782.740217 Validation Loss: 1091.573996
Validation loss decreased (1115.092531 --> 1091.573996). Saving model ...
Epoch 11/20
Epoch: 11 Training Loss: 3652.762325 Validation Loss: 1084.334752
Validation loss decreased (1091.573996 --> 1084.334752). Saving model ...
Epoch 12/20
Epoch: 12 Training Loss: 3631.786594 Validation Loss: 1110.550763
Epoch 13/20
Epoch: 13 Training Loss: 3594.383026 Validation Loss: 1090.805948
Epoch 14/20
Epoch: 14 Training Loss: 3594.376469 Validation Loss: 1095.772802
Epoch 15/20
Epoch: 15 Training Loss: 3591.559703 Validation Loss: 1071.639330
Validation loss decreased (1084.334752 --> 1071.639330). Saving model ...
Epoch 16/20
Epoch: 16 Training Loss: 3498.333718 Validation Loss: 1084.942226
Epoch 17/20
Epoch: 17 Training Loss: 3481.485294 Validation Loss: 1086.111507
Epoch 18/20
Epoch: 18 Training Loss: 3531.934473 Validation Loss: 1060.883076
Validation loss decreased (1071.639330 --> 1060.883076). Saving model ...
Epoch 19/20
Epoch: 19 Training Loss: 3532.836403 Validation Loss: 1048.723871
Validation loss decreased (1060.883076 --> 1048.723871). Saving model ...
Epoch 20/20
Epoch: 20 Training Loss: 3503.658860 Validation Loss: 1060.619535

(IMPLEMENTATION) Test the Model

Try out your model on the test dataset of landmark images. Use the code cell below to calculate and print the test loss and accuracy. Ensure that your test accuracy is greater than 60%.

Some reading: https://discuss.pytorch.org/t/passing-to-the-optimizers-frozen-parameters/83358

Step 3: Write Your Landmark Prediction Algorithm

Great job creating your CNN models! Now that you have put in all the hard work of creating accurate classifiers, let's define some functions to make it easy for others to use your classifiers.

(IMPLEMENTATION) Write Your Algorithm, Part 1

Implement the function predict_landmarks, which accepts a file path to an image and an integer k, and then predicts the top k most likely landmarks. You are required to use your transfer learned CNN from Step 2 to predict the landmarks.

An example of the expected behavior of predict_landmarks:

>>> predicted_landmarks = predict_landmarks('example_image.jpg', 3)
>>> print(predicted_landmarks)
['Golden Gate Bridge', 'Brooklyn Bridge', 'Sydney Harbour Bridge']

(IMPLEMENTATION) Write Your Algorithm, Part 2

In the code cell below, implement the function suggest_locations, which accepts a file path to an image as input, and then displays the image and the top 3 most likely landmarks as predicted by predict_landmarks.

Some sample output for suggest_locations is provided below, but feel free to design your own user experience!

(IMPLEMENTATION) Test Your Algorithm

Test your algorithm by running the suggest_locations function on at least four images on your computer. Feel free to use any images you like.

Question 4: Is the output better than you expected :) ? Or worse :( ? Provide at least three possible points of improvement for your algorithm.

Answer: (Three possible points for improvement)

Better. I suspected the red water and random crowd in the Trevi Fountain picture might throw off the CNN, but it appeared to be color invariant in this example image. Likewise for the Seattle Japanese Garden (SJG), I tried to find difficult-to-classify photos of it. The CNN struggled with the detailed door photo of the SJG, and the image of the turtles could have been taken just about anywhere. If turtles only ever showed up in the training dataset for the SJG, and no other landmarks, then it's entirely possible with enough training photos the CNN would begin to classify pictures with turtles as belonging to the SJG.

  1. Add more training photos. Specific fauna and flora local to each landmark, but without necessarily showing the landmark itself in the photo.
  2. Train for more epochs. In this project I ran the ResNet50 for 20 training epochs, but I feel with a longer baking time the CNN could be improved.
  3. Add further transforms to the input images, for example simulate different lighting conditions by modifying brightness, or colors. Or simulate an out-of-focus camera, by adding a blurring transform. The TrivialAugmentWide, RandAugment, GaussianBlur and ColorJitter transforms found here could do the trick: https://pytorch.org/vision/main/auto_examples/plot_transforms.html#sphx-glr-auto-examples-plot-transforms-pyhttps://pytorch.org/vision/main/auto_examples/plot_transforms.html#sphx-glr-auto-examples-plot-transforms-py